Papers with attention modules
MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Models (2026.findings-acl)
Copied to clipboard
Junjie Ye, Caishuang Huang, Zhuohan Chen, Wenjie Fu, Chenyuan Yang, Leyi Yang, Yilong Wu, Peng Wang, Meng Zhou, Xiaolong Yang, Tao Gui, Qi Zhang, Zhongchao Shi, Jianping Fan, Xuanjing Huang
| Challenge: | Existing research has focused on constraint categories, offering little guidance for improving instruction following abilities. |
| Approach: | They propose a multi-dimensional constraint framework that allows for instruction following . they construct 9,106 code-verifiable samples and evaluate 18 LLMs . |
| Outcome: | The proposed framework improves instruction following performance without compromising general performance. |
Towards Explaining Subjective Ground of Individuals on Social Media (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale language models have been reducing the gap between machines and humans in understanding how the real world works, yet understanding an individual’s perspective and behavior from text is far from being resolved. |
| Approach: | They propose a neural model that learns subjective grounds of individuals and accounts for their judgments on situations posted on social media. |
| Outcome: | The proposed model provides human-readable explanations of an individual’s subjective preference in judging social situations and also learns an individual's orientation towards abstract moral concepts. |
DREEAM: Guiding Attention with Evidence for Improving Document-Level Relation Extraction (2023.eacl-main)
Copied to clipboard
| Challenge: | Document-level relation extraction (DocRE) is a task of identifying relations between entities in a document. evidence retrieval (ER) in DocRE faces two major issues: high memory consumption and limited availability of annotations. |
| Approach: | They propose a memory-efficient approach that uses evidence as the supervisory signal . they propose er self-training to learn ER from automatically-generated evidence . |
| Outcome: | The proposed method exhibits state-of-the-art performance on the DocRED benchmark . it uses evidence as the supervisory signal and self-trains on massive data without annotations . |
Dual Attention Networks for Visual Reference Resolution in Visual Dialog (D19-1)
Copied to clipboard
| Challenge: | Visual dialog (VisDial) requires a dialog agent to answer a series of questions grounded in an image. |
| Approach: | They propose dual attention networks (DAN) for visual reference resolution in VisDial. |
| Outcome: | The proposed model outperforms the previous state-of-the-art model on VisDial datasets. |
Selective Attention for Context-aware Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Recent work in context-aware NMT considers only a few previous sentences as context . current systems fail to achieve fluent, good quality translation for a full document . |
| Approach: | They propose a top-down approach to hierarchical attention for context-aware NMT which uses sparse attention to selectively focus on relevant sentences in the document context. |
| Outcome: | The proposed approach outperforms context-agnostic baselines and context-based baselines on English-German datasets. |
Attention is not Explanation (N19-1)
Copied to clipboard
| Challenge: | Attention mechanisms have seen wide adoption in neural NLP models. |
| Approach: | They perform extensive experiments to assess the degree to which attention weights provide meaningful "explanations" they find that attention weighted inputs are often uncorrelated with gradient-based measures of feature importance . |
| Outcome: | The proposed model is based on a distribution over attended-to input units . the findings show that attention weights are often uncorrelated with features . |
HIT - A Hierarchically Fused Deep Attention Network for Robust Code-mixed Language Representation (2021.findings-acl)
Copied to clipboard
| Challenge: | linguistics and morphology of resource-short code-mixed texts remain a key challenge in text processing. |
| Approach: | They propose a hierarchical transformer-based framework that captures the semantic relationship among words and hierarchically learns sentencelevel semantics using a fused attention mechanism. |
| Outcome: | The proposed framework improves on one European and five Indic languages on four NLP tasks on eleven datasets. |
Leveraging Locality in Abstractive Text Summarization (2022.emnlp-main)
Copied to clipboard
Yixin Liu, Ansong Ni, Linyong Nan, Budhaditya Deb, Chenguang Zhu, Ahmed Hassan Awadallah, Dragomir Radev
| Challenge: | Neural attention models have improved on many natural language processing tasks, but their quadratic memory complexity hinders their applications in long text summarization. |
| Approach: | They propose to use a restricted context to study locality in text summarization . they propose to employ a quadratic memory growth with respect to the input length . |
| Outcome: | The proposed model has better performance than baseline models with efficient attention modules. |
Guided Attention Multimodal Multitask Financial Forecasting with Inter-Company Relationships and Global and Local News (2022.acl-long)
Copied to clipboard
| Challenge: | Stock returns in financial markets are influenced by textual information from diverse sources. |
| Approach: | They propose a model that captures both global and local multimodal information for investment and risk management-related forecasting tasks. |
| Outcome: | The proposed model outperforms state-of-the-art models in several forecasting tasks and important real-world applications. |
Integrating Vectorized Lexical Constraints for Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies focus on integrating discrete lexical constraints into neural machine translation models. |
| Approach: | They propose to integrate constraints into NMT models by integrating them into keys and values . they show that their method outperforms representative baselines on four language pairs . |
| Outcome: | The proposed method outperforms baselines on four language pairs, showing superiority . |
Analyzing Memorization in Large Language Models through the Lens of Model Attribution (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing research has focused on extracting memorized content from LLMs or developing memorization metrics without exploring the underlying architectural factors that contribute to memorizing. |
| Approach: | They analyze how attention modules at different layers impact its memorization and generalization performance by using attribution techniques. |
| Outcome: | The proposed model can be used to mitigate memorization while keeping other components like layer normalization and MLP transformations intact. |
Differentiable Window for Dynamic Local Attention (2020.acl-main)
Copied to clipboard
| Challenge: | Existing general purpose components for learning differentiable windows are hard to optimize. |
| Approach: | They propose a new neural module and general purpose component for dynamic window selection that can enable more focused attentions over the input regions. |
| Outcome: | The proposed approach improves on a myriad of NLP tasks including machine translation, sentiment analysis, subject-verb agreement and language modeling. |
CHESS: Optimizing LLM Inference via Channel-Wise Thresholding and Selective Sparsification (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for activation sparsification do not capture the relationship between activation and model performance. |
| Approach: | They propose a general activation sparsification approach using channel-wise thresholding and selective sparsifying to capture the relationship between activation and model performance. |
| Outcome: | The proposed approach reduces the number of activated neurons during inference by 1.27x over eight downstream tasks while activating fewer parameters than existing methods. |
Do All Autoregressive Transformers Remember Facts the Same Way? A Cross-Architecture Analysis of Recall Mechanisms (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has identified MLP modules in early layers as key contributors to factual recall. |
| Approach: | They conduct a comprehensive evaluation of factual recall across several models . they find that Qwen-based models behave differently from previous patterns . |
| Outcome: | The results show that attention modules in the earliest layers contribute more to factual recall than MLP modules. |